Parallelization of a finite difference scheme for solving systems of 2D Sine-Gordon equations

Systems of perturbed 2D Sine-Gordon equations coupled via a cyclic tridiagonal matrix are solved numerically by a second-order centered finite difference scheme. The systems are considered on rectangular domains. First an OpenMP parallel program is realized and very good performance scalability inside one computational node is achieved. The tests on one computational node of the CPU platform in the IICT-BAS cluster, Bulgaria, show speedup about 11 and on the CPU part of one node in the HybriLIT cluster, JINR, Russia, show speedup about 30. After that the OpenMP parallelization effect is multiplied by a fully hybrid strategy: one MPI process per cluster node and OpenMP on the cores of the node. The Hybrid MPI+OpenMP parallelization is realized as follows. Domain decomposition is made and ghost points are introduced to support receives between MPI processes. Inside a subdomain the work is shared between OpenMP threads. At every mesh point a system with one and the same cyclic tridiagonal matrix is solved. Solving the system with this matrix is the smallest piece of work distributed between threads. Only master threads communicate outside the parallel OpenMP region with ordered MPI sends and receives. Computations on 12 nodes of the CPU platform in the IICT-BAS cluster show significant speedup up to 116.